…ory destinations for different processes

…ifications to P3 module to include more runtime switches, nsmall checks, etc.

Instrumentation (debug; can be stripped before merging to main code):
- p3_chksum: PRE/POST bitwise field checksums per (itimestep, j-slab) to
  per-rank p3_chksum_NNNN.out (bitsum/bitsum2 + chksum_write in
  module_mp_p3.F). Covers qv,th,qc,nc,qr,qnr,w,p + all per-category ice
  fields. Labels use true model j via j_index (now passed at all P3 call
  sites in mp_driver.F).
- p3_chkpts: per-point raw 32-bit patterns with local AND full-grid (i,j,k)
  to p3_chkpts_NNNN.out, for exact divergent-point isolation (LARGE output;
  restart-bracketing runs only). Global offsets myi1/myj1 via param.F.
- compare_p3_chksum.py: compares two runs' files (both formats
  auto-detected) under an ice-category permutation (--swap/--map); single
  rank or whole simulation with earliest-divergence summary.

Feature (candidate keeper):
- qsum_invariant namelist option: permutation-invariant (sorted) sums of
  moist scalars in the prognostic dynamics path (solve1, solve2, turb;
  sortsumq in input.F), for bit-identity testing under category relabeling.

Outcome (NOTES.md "RESOLVED"): swap divergence originates inside p3_main
via order-of-operations roundoff (audit sec. 3), seeding at the first
timestep the swapped slots' contents overlap anywhere -- advection
undershoot halos suffice. Host dynamics exonerated; no sec. 2-class bug
implicated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

… census findings

Three independent A/B switches in &p3_params, each making one candidate
order-sensitive category reduction in p3_main (practically) invariant via
double-precision accumulation rounded once, default paths untouched:
- p3_clipdp: entry clipping pass qv/th add-back (now DEPRIORITIZED --
  exonerated by the t=175 onset census: no qv flips at any of the 158
  divergent points, and a clip order-flip is necessarily a 1-ulp qv change)
- p3_epsdp:  epsi_tot/epsiw_tot accumulation over ice categories
- p3_limdp:  conservation-limiter category sums (sum(qidep)+sum(qinuc),
  sum(qisub), cloud/rain/vapor sinks/sources)

Findings recorded in NOTES.md (HANDOFF section added for session
continuity): onset is 158 points in one anvil blob at itimestep=175, ~85
columns seeding independently; source-point PRE decode shows real slot-3/4
coexistence with comparable masses but number concentrations 6 orders
apart, putting the category sums in the near-ulp flip-prone regime.
Remaining discriminating experiment: p3_epsdp vs p3_limdp, one at a time,
both members of the swap pair.

Also: decode_point.sh helper (per-point dump record -> named float values)
added to repo root.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…ch nulls

Single-switch A/B runs (p3_epsdp alone, p3_limdp alone) left the divergence
onset unchanged (same timestep/point/fields; only downstream counts differ).
Two lessons recorded in NOTES.md: the one-at-a-time protocol cannot isolate
the seed when multiple reductions flip the same points, and sum(nitot) in
deposition nucleation (live: Cooper, dest cat 1) was unswitched in all runs
-- its flip propagates to all categories through the binding satadj limiter
ratio, which p3_limdp does not protect against.

p3_nucdp makes that sum order-invariant (computed once, double precision).
Next experiment: ALL four invariance switches enabled together in both
members of the swap pair; onset unchanged would exclude all known
order-of-operations sites and leave stale per-category state (NaN-poison +
FP-trap hunt) as the remaining explanation.

Also noted in NOTES.md: latent p3_init bug -- optional iparamice_in is
unconditionally overwritten by the default (iparamice=1) set after it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…l known sums

The combined run (all four order-invariance switches enabled in both swap
members) left the divergence onset unchanged -- every known sec.-3
order-of-operations reduction is excluded as the seed. Remaining
explanation: sec.-2-class stale per-category state inherited across
iice_loop1 iterations.

p3_nanpoison poisons all per-category work variables at the top of every
iice_loop1 iteration: f1pr01-38 lookup values, interpolation weights
(dum1/3/4/5/6/7), rhop, mu_i as signaling NaNs (bit pattern via transfer,
set in p3_init), and lookup indices (dumi/dumj/dumii/dumjj/dumll/dumzz) as
huge sentinels. Consumption before being set in the same iteration traps
under -ffpe-trap=invalid / -fcheck=bounds. Deliberate per-iteration
defaults (Eii_fact, tmp1) are not poisoned.

Verified on this machine (arm64 macOS): gfortran's -ffpe-trap fires
(EXC_BAD_INSTRUCTION) but its handler loops instead of terminating; running
under lldb stops at the exact consuming line. Debug-build recipe and lldb
workflow documented in NOTES.md; single serial run suffices (no A/B).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…FP-trap run)

The mass/area-weighted effective-radius average in mp_p3_wrapper_wrf divided
unconditionally; at ice-free points all weights are zero, so diag_effi_ave
has been silently NaN there in every nCat>=2 run (harmless with radopt=0,
garbage in output, would poison radiation if enabled). The division is now
guarded and the 25-micron default retained where no ice contributes.

First find of the stale-state hunt; benign for the divergence question but
had to be silenced for the trap run to proceed past the first call.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…gories

Second find of the p3_nanpoison FP-trap run: the zidep compound conditional
evaluated qiliq/qitot even when epsiw(iice)=0 (Fortran .and. does not
short-circuit), giving 0/0 at empty categories -- NaN in a comparison,
benign in effect (compares false, matching intent) but an invalid op every
time. Nested the ifs, same pattern as the existing ERM splits.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Found by the p3_nanpoison FP-trap hunt (trap at the dumden=f1pr16
consumption): iice_loop2 computed the pre-update 3-moment mu_i for every
category using the scalar f1pr16 -- the mass-weighted mean density of
whichever category LAST ran the lookup in iice_loop1. Wrong species'
density for all but one category, and which species leaked depends on
category slot order: precisely what the destination-category swap changes,
seeding divergence wherever two ice categories coexist. mu_i_s feeds the
post-update zitot adjustment, matching the observed onset (ice-moment-only
flips, incl. a pure-zitot record; immune to all order-invariance switches).

Fix: per-category f1pr16_s(nCat), saved inside each category's own lookup
block (-1 = lookup not accessed this step); iice_loop2 uses f1pr16_s(iice)
and retains mu_i_initial when this category's lookup did not run. This is
a physics fix affecting all 3-moment nCat>1 runs, not merely a
reproducibility fix.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…restructure

Confirmation runs complete the empirical hierarchy: (1) stale f1pr16 bug
(fixed) was the dominant seed; (2) with it fixed, the category-sum
reductions seed at first coexistence (suppressed by the dp switches);
(3) with those invariant too, the onset finally moves (t=175 -> t=176)
and lands as a lone-qr flip -- the iice_loop3 sequential shared-field
update fingerprint, the remaining inherent floor.

NOTES.md now carries the closed investigation record, the
upstream-reportable bug list, and a NEXT PROJECT spec (scope, approach,
acceptance tests, opening prompt) for implementing permutation-invariant
tendency application in a fresh session.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…builds

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Swapping ice-category destinations (cats 3<->4) now produces bit-identical
results (swap pair identical every step through t=200, matching the no-swap
control). Completes the permutation-invariance program by eliminating the
remaining fixed-slot-order category reductions:

- module_mp_p3.F: §4(c) restructure of p3_main -- iice_loop3 shared-field
  tendencies (qc/nc/qr/nr/qv/th) and the post-loop3 clip loops now accumulate
  per-category increments in double precision and apply once; the former
  clipdp/epsdp/limdp/nucdp double-precision paths are made permanent
  (unconditional). Ice-ice interaction transfers + clip/densify restructured to
  be invariant-ready (exact no-op while disabled).
- turb.F (calcnm): total-water qt sum made category-order-invariant via sortsumq
  (host turbulence seed: qt -> thv -> N^2 -> subgrid TKE -> khh/khv -> tendency).
- module_mp_p3.F: Hallett-Mossop nimul/qrmul accumulation into the (swapped)
  destination category now summed in double precision, applied once.

Cleanup: removed the obsolete diagnostic switches p3_clipdp/epsdp/limdp/nucdp
(now permanent) and p3_nanpoison (stale-state hunt complete) from
module_mp_p3.F, param.F and run/namelist.input. Kept qsum_invariant and the
p3_chksum/p3_chkpts checksum instrumentation. All temporary debug probes removed.

Requires qsum_invariant=.true. for full host-side invariance. Full seed list,
upstream-reportable bugs, and the divergence-diagnosis workflow are recorded in
NOTES.md.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…in code (or setting p3_nimax to -1 in namelist). Namelist setting overrides in-code setting.

Four new P3 configurations, using the next free ptype numbers so existing
ones are untouched:

  64 = as ptype 57 (3-moment ice, no liqFrac) but 5 ice categories
  65 = as ptype 57 but 6 ice categories
  66 = as ptype 63 (3-moment ice, liqFrac)    but 5 ice categories
  67 = as ptype 63 but 6 ice categories

P3 itself was already generic in nCat (allocatable per-category arrays,
deltaD_init already has cases for 5 and 6+, and lookupTable_2 is indexed by
size/rime/density bins rather than by category), so the work was in the
category-destination dispatch and the CM1-side plumbing:

- module_mp_p3.F: catdest_*_n5 / _n6 added for all six ice-initiation
  mechanisms (module vars, p3_init optional arguments with clamping, and
  nCat==5/6 branches at each of the six iice_dest dispatch sites). Without
  these, cat_init_new=.true. would have left iice_dest unset at nCat>4.
  Defaults continue the progressive split of the _n2..._n4 series so no
  category lacks a source: at nCat=5 deposition nucleation moves to its own
  category, at nCat=6 homogeneous rain freezing moves to its own.
- module_mp_p3.F: mp_p3_wrapper_wrf extended with qit/qni/qir/qib/qzi/qli_5
  and _6, diag_vmi/dmi/rhoi_5,_6 and diag_dhmax_5,_6, plus the corresponding
  gather/scatter blocks. diag_effi_ave's hard-coded nCat<=4 expressions
  replaced by a sequential loop over categories; the accumulation order is
  the same one the explicit sums used, so nCat<=4 is unchanged bit-for-bit.
- param.F: p3_catdest_*_n5/_n6 namelist variables + MPI_BCAST, p3_init
  dispatch branches, ptype range tests extended to 67, and the
  numq/qname/qunit/qmag/cloudvar/np3a/np3o blocks for the four new ptypes.
- mp_driver.F: four new per-ptype blocks (p3a pack, wrapper call at
  n_iceCat=5/6, q3d unpack, p3o and out2d/out3d fills).
- writeout.F: ptypes 64-67 added to the category 1-4 P3 diagnostic blocks,
  plus new blocks for categories 5 and 6 (p3_vmi5/6, p3_dmi5/6, p3_rhoi5/6,
  p3_dhmax5/6).
- radiation_driver.F: ptypes 64-67 added to the effc/effi list.
- README.namelist: ptype list now documents 54, 56, 57 and 60-67 (it had
  only covered 50-53) and explains the p3_catdest_*_nN suffix convention.
- run/namelist.input: the twelve new _n5/_n6 catdest keys.

Verified: full build clean; all four ptypes run with the expected
numq/nqs2/qname tables; a 2400 s ptype-65 run populates all six categories
(qi5 via deposition nucleation, qi6 via homogeneous rain freezing) and emits
all 24 per-category diagnostics. Regression: a 2400 s ptype-57 run is
bit-identical (cm1out_s/u/v/w/stats) against a binary built from the parent
commit.

Note: older executables will reject a namelist containing the new _n5/_n6
keys, so A/B runs against an older binary need them stripped.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

CLAUDE.md records the things that are non-obvious from the tree: that
src/Makefile is the active build config and needs NETCDFC_DIR/NETCDF4_DIR/
HDF5_DIR in the environment, that src/*.f90 and run/onefile.F are generated
and must not be edited, the P3 ptype table and the set of files a new
configuration touches, the run-directory lookup-table requirements, and a
pointer to NOTES.md plus the surviving p3_chksum/p3_chkpts tooling.

.gitignore: .claude/settings.local.json holds the netCDF/HDF5 paths for
whichever machine the checkout lives on, so it stays untracked.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…ed swap variance

NOTES.md had not been touched since 11300ba and did not reflect four
subsequent commits or this session's work. Added a dated section at the top
(the file is newest-first) covering:

- The new ptypes 64-67 (5- and 6-ice-category configurations), what had to
  change to support nCat>4, and the verification performed. Flags that the
  _n5/_n6 catdest defaults are a science choice needing review, and that
  ptype 64 is therefore not literally "ptype 57 with an inert 5th category".
- The deferred swap-variance investigation: recorded as reported (smaller
  than the pre-fix divergence, cause unknown, log_iceice_interaction the
  first hypothesis rather than an established cause) so it is not lost.
- The 07-21..07-25 commits: the Bell Makefile, the NSSL3M_Feb_2025 merge,
  log_maxnsmall, and the per-category max-Ni limiter (max_total_Ni ->
  max_Ni, impose_max_Ni, p3_nimax = -1 to disable).
- That log_iceice_interaction now defaults .true., which the 07-20
  bit-identity result did not cover -- the restructured cross-category
  path is invariant-ready but has never been verified under a swap test.

Also flags three DTD-authored P3 changes that the NSSL3M merge reverted
(the pre-P3 pdefq loops, dbrk 3000e-6 -> 600e-6, and the coll_3mom_LF rain
gate nsmall -> 0, the last of which runs against the 2.1 gate-alignment
work) as needing confirmation that the reversion was intended.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The 07-27 NOTES entry attributed the reverted scaled-mean-rain-size gate
(nr.gt.nsmall -> nr.gt.0) to access_lookup_table_coll_3mom_LF. It is
actually in find_lookupTable_indices_1b (module_mp_p3.F ~12354); the name
came from a git diff hunk header, whose Fortran function detection had
latched onto a much earlier subroutine.

Also drops the claim that this revert runs against the 2.1 gate-alignment
work. It does not: 2.1 moved the process-block and sedimentation gates onto
nsmall, and this lookup-index gate was not among them. Reframes all three
NSSL3M-merge reverts as deliberate, per DTD.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Grid-point noise in ptype-54 runs traces to a single pattern: the host
delivers a hydrometeor species with mass but no number, and P3 manufactures
the missing number (nsmall) rather than declining to act.  That implies an
absurd mean particle size, which the lookup-table index clamp then silently
absorbs, so the failure is laundered into a legal table index and never
reported.  Confirmed independently for ice, rain and cloud; the root cause is
upstream, in the host's transport of number concentration, and is untouched
here.

Two temporary A/B switches, both defaulting to existing behaviour:

  p3_nifloor   .false. disables every nitot/nc floor (21 guarded sites)
  p3_nrfloor   .false. disables every nr floor      (15 guarded sites)

With a floor off, the affected point keeps its mass and is rejected by
nitot/nc/nr conditions on the downstream gates instead of running at an
absurd size.  Both exit clips were restructured so that their mass-clearing
else branch still fires only on q < qsmall -- attaching the number condition
to the outer test would relocate the clipping rather than remove it.

Every floor is now owned by the switch for the variable it floors, including
the log_maxnsmall floors.  A floor sitting under a gate that tests a
different variable silently breaks the invariant the downstream gates depend
on; one such line produced -Inf + Inf and a hard fault.

Instrumentation, following the existing p3_chksum/p3_chkpts pattern:

  p3_limlog, p3_limlog_rat    per-firing log of every point-local limiter
  p3_dbg_i/j/k, p3_dbg_halo   targeted point dump with phase checkpoints

Also fixes an out-of-bounds access in mp_driver.F: qunit(lccn) was evaluated
with lccn = 0 whenever p3_ccn_on = 0, and in an optimised build the guarded
branch would then read and write q3d(i,j,k,0), a full 3-D slab before the
array.  Never A/B'd against an unpatched binary, so whether it was firing in
production runs is unknown.

Full record, including the wrong turns and the open items, is in
p3_smallvalue_gating_notes.md.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…itch